
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
•Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



10/693,295 



21876 



10/24/2003 



7590 



10/04/2006 



FISH & RICHARDSON P.C. 
P.O. Box 1022 

MINNEAPOLIS, MN 55440-1022 



Stephen N. Schiller 



07844-616001 / P573 



1482 



EXAMINER 



ALLISON, ANDRAES 



ART UNIT 



PAPER NUMBER 



2624 



DATE MAILED: 10/04/2006 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 


Application No. 

10/693,295 


Applicant(s) 
SCHILLER ET AL 


Examiner 

Andrae S. Allison 


Art Unit 

2624 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communioation. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after t he mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

Responsive to communication(s) filed on October 24. 2003 . 
2a)D This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) E3 Claim(s) 1-32 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) S Claim(s) 1-32 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) S The specification is objected to by the Examiner. 

10)13 The drawing(s) filed on January 3. 2005 is/are: a)S accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 !)□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1. Q Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) Notice of References Cited (PTO-892) 

2) CD Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) CD Information Disclosure Statement(s) (PTO/SB/08) 

Paper No(s)/Mail Date . 



4) □ Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) □ Notice of Informal Patent Application 

6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 08-06) 



Office Action Summary 



Part of Paper NoVMail Date 20060905 



Application/Control Number: 10/693,295 
Art Unit: 2624 



Page 2 



DETAILED ACTION 

Specification 

1 . The disclosure is objected to because of the following informalities: 

a. The phrase "pixel of inertest" on page 6, lines 11-12 and line 1 3 should 
read "pixel of interest" because the word "interest" is misspelled. 

b. The phrase "two input-" on page 16, line 16 should read "two input" 
because the dashes "- -" should be deleted. 

c. The phrase "gating node- to" on page 16, line 1 7 should read "gating s 
node to"" because the dashes "- -" should be deleted. 

Appropriate correction is required. 



Claim Rejections - 35 USC §112 

2. The following is a quotation of the second paragraph of 35 U.S.C. 1 1 2: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 



3. Claims 17 and 18 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 
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Claim 17 recites the limitation "the neighborhood of pixels" in line 2. 
There is insufficient antecedent basis for this limitation in the claim. It is 
believed claim 1 7 was intended to depend on claim 2. 

Claim 18 is being rejected as incorporating the deficiencies of claim 17 upon 
which it depends. 

Appropriate correction is required. 

Claim Rejections - 35 USC § 101 

4. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

Claims 31 and 32 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter as follows. Claims 31 and 32 
recites the limitation "machine readable medium" in line 1 of the above claims. Due to 
published Interim Guidelines "machine readable medium" is no longer considered as 
statutory. Therefore the limitation "machine readable medium" should be changed to 
"computer readable medium". 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
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the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 1-3, 5-7, 13-16, 20, 25, 31 and 32 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Luo et al (US Patent No.: 6,504,951) in view of Norton (US 
Patent No.: 5,912,994). 

As to claim 1 , Luo discloses a method for defining a boundary (differentiate true 
sky regions from other similarly color and textured subject matters; column 4, lines 33- 
34) separating a first region (e.g. non-sky-colored pixels, column 5, line 41) and a 
second region (e.g. sky colored pixels; column 5, line 41) of a digital image (200, Fig.2), 
the digital image including one or more color arrangements characteristic (see Fig 12 A, 
where the first region includes water and a boat which has various colors) of the first 
region and one or more color arrangements characteristic (sky color which includes 
non-uniform blue and almost white colors; column 6, lines 59-65) of the second region. 
Lou teaches determining using a learning machine (color classifier, column 8, line 47) 
based on one or more of the color arrangements, which pixels of the image satisfy 
criteria (if gradient value is above predetermined threshold, the region is considered a 
non-sky regions; column 6, lines 18-19) for classification as associated with the first 
region. Luo also teaches determining using a learning machine (color classifier, column 
8, line 47), based on one or more of the color arrangements, which pixels of the image 
satisfy criteria (if belief value of candidate region is above a certain threshold, the 
regions is declared a sky column 6, lines 18-19) for classification as associated with the 
second region. Luo teaches identifying pixels of the image that are determined not to 
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satisfy the criteria for classification as being associated either with the first region or the 
second region (marginal pixels; column 10, line 53). 

However, Luo does not teach identifying pixels and decontaminating the 
identified pixels to define a boundary between the first and second regions. Norton 
teaches a method of masking an object (column 1, line 59) including identifying pixels (a 
pixel is identified if it in a certain range, column 16, lines 45-50) and decontaminating 
the identified pixels to define a boundary between the first and second regions (column 
17, lines 1-4). Therefore it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to have added the method of masking an object to the 
method of defining a boundary of Luo for detecting a boundary of a substantially color- 
homogeneous region of a frame of a digitized picture stock (column 1, lines 61-65) with 
"increased efficiency" (column 1 , line 54). Also, the process of Norton allows for better 
segmentation of the various regions of the image and better identification of ambiguous 
pixels. 

As to claim 31, this claim differ from claim 1 only that the limitations a 
computer program product, tangibly stored on machine readable medium and the 
product comprising instructions operable to cause a processor are additively recited in 
the preamble. Lou clearly teaches a computer program product (column 3, line 6). Note 
the discussion above, Norton teaches machine-readable medium (memory, column 3, 
line 24) and the product comprising instructions operable to cause a processor (12, see 
Fig 1). 
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As to claim 32, Luo teaches a computer program product (column 3, line 6), 
tangibly stored on machine readable medium, for segmenting a first region (e.g. non- 
sky-colored pixels, column 5, line 41) and a second region (e.g. sky colored pixels; 
column 5, line 41), each region including one or more color arrangements (see Fig 12 a) 
that are characteristic of the region. However, Lou does not teach machine readable 
medium and the product comprising instructions operable to cause a processor to: 
receive an input that selects a portion of the first region and an input that selects a 
portion of the second region; identify pixels, based on the inputs and the color 
arrangements of the first and second regions, located in the first region; and identify 
pixels, based on the inputs and the color arrangements of the first and second regions, 
located in the second region. 

Norton teaches a method of masking an object (column 1 , line 59) including 
machine readable medium (memory, column 3, line 24) and the product comprising 
instructions operable to cause a processor (12, see Fig 1) to: receive an input that 
selects a portion of the first region and an input that selects a portion of the second 
region (202, see Fig 6); identify pixels (210, see Fig 6) based on the inputs and the color 
arrangements of the first and second regions, located in the first region; and identify 
(210, see Fig 6) pixels, based on the inputs and the color arrangements of the first and 
second regions, located in the second region. Therefore it would have been obvious to 
one of ordinary skill in the art at the time the invention was made to have added the 
method of masking an object to the method of defining a boundary of Luo for detecting 
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a boundary of a substantially color-homogeneous region of a frame of a digitized picture 
stock (column 1, lines 61-65) with "increased efficiency" (column 1 , line 54). Also, the 
process of Norton allows for better segmentation of the various regions of the image 
and better identification of ambiguous pixels. 

As to claim 2, Luo teach wherein a pixel being considered by the learning 
machine is associated with a corresponding neighborhood of pixels, the method further 
comprising: providing to the learning machine input information specifying a color 
arrangement of the corresponding neighborhood of pixels, wherein the learning 
machine is configured to classify the pixel being considered, based on the color 
arrangements of the corresponding neighborhood of pixels, as either associated with 
the first region or with the second region (see column 10, lines 53-60 where the 
marginal pixels mentioned above are relabeled base on the neighboring pixels). 

As to claim 3, Lou teaches a method wherein a color arrangement represents a 
visual texture (e.g. the water in Fig 12 A). 

As to claim 5, Luo teaches a method wherein the learning machine is a neural 
network (column 5, lines 4-5) 

As to claim 6, Luo teaches a method wherein the learning machine is configured 
to provide an output classifying the pixel being considered, the output indicating a 
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probability of the pixel being associated with the first region and a probability of the pixel 
being associated with the second region (see column 8, lines 47-50, where a pixel 
classifier is trained to output a belief value between 0 and 1 a pixel, 1 indicating a highly 
likely to be blue sky and 0 indicating not very likely to be blue sky). 

As to claim 7, Lou teaches a method wherein the output is a floating point 
(number between a lower number (e.g. 0; column 8, line 45) and an upper number (e.g. 
1 ; column 8, line 45) the lower number indicating a one-hundred percent probability of 
the pixel being associated with the second region (e.g. 0 indicating the likelihood a pixel 
as non-sky; column 8, line 45) and the upper number indicating a one-hundred percent 
probability of the pixel being associated with the first region (e.g. 1 indicating the 
likelihood a pixel ideal blue sky; column 8, line 45). 

As to claim 13, Luo teaches a method of training the learning machine to classify 
pixels (column 8, lines 20-21). 

As to claim 14, Lou teaches a method wherein training includes selecting, based 
on user input, a training set of pixels used to train the neural network (column 8, lines 
30-35). 

As to claim 15, note the discussion above, Norton teaches the method wherein 
the training set of pixels selected includes pixels located within a particular range of the 
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boundary (see Fig 6, step 208, where a range can be adjusted). 

As to claim 16, note the discussion above, Norton teaches a method wherein the 
particular range is 20 pixels from either side of the boundary. In the spec (page 19, 
[p][22], line 10), Applicant states that the particular range can in the range of 2 to 100 
pixels. Consequently, there is no significant advantage or disclosed criticality in 
choosing the particular range to be 20 pixels, therefore it would have been obvious that 
the particular range can be 20 pixels. 

As to claim 20, note the discussion above, Norton teaches constructing from the 
identified pixels a boundary mask that indicates which pixels of the digital image are the 
identified pixels (column 17, lines 16-21). 

As to claim 25, note the discussion of claim 2 above. 

7. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Luo et al 
(US Patent No.: 6,504,951) in view of Norton (US Patent No.: 5,912,994) further in view 
of Wang et al (Pub No.: US 2003/0007683). 

As to claim 4, Lou in view of Norton does not teaches a method wherein the 
learning machine is a support vector machine. Wang teaches a method of separating 
text from drawings (page 1, [p][0005], lines 1-2) that includes using a support vector 
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machine as a learning machine (page 1 , [p][0006], lines 4). Therefore it would have 
been obvious to one of ordinary skill in the art at the time the invention was made to 
have substituted the color classifier of Luo with the support vector machine of Wang for 
classifying a stroke according to the slope curvature vector as either "text" or "unknown" 
(page 1, [p][0006], lines 8-10), because each reference reg i on is for segmenting out one 
region from another and the use of a support vector machine is well known to segment 
regions of an image. 

8. Claims 8-12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Luo 
et al (US Patent No.: 6,504,951) in view of Norton (US Patent No.: 5,912,994) further in 
view of Littmann et al (NPL Document titled "Apative Color Segmentation - A 
comparison of Neural and Statistical Methods). 

As to claim 8, Lou teaches wherein the lower number is -1 and the upper number 
is 1 (column 8, line 45). However Both Lou and Norton do not teach wherein the lower 
number is -1. Littmann teaches a method of comparing learning classifiers (page 175, 
[p][2], line 4-6) including wherein the lower number is -1 (page 176, [p][3], line 7). 
Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have added the method of comparing classifier of Littmann to the 
method of defining a boundary of Luo as modified by Norton for classifying pixels in an 
image as foreground (e.g. hand, page 175, [p][4], line 6) and background (page 175, 
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[p][4], line 6), and because this simply gives a commonly used value to represent one 
of the conditions. 

As to claim 9, note the discussion above, Littmann teaches a method of 
converting the floating point number to an integer between a first integer and a second 
integer, the first integer indicating a one-hundred percent probability of the pixel being 
associated with the second region, and the second integer indicating a one-hundred 
percent probability of the pixel being associated with the first region (see page 179, 
[p][7], lines 7- 8, where the output the classifier are scale to gray values). 

As to claim 10, note the discussion above, Littmann teaches a method wherein 
the first integer is 0 and the second integer is 256 (see page 1 79, [p][7, lines 8-9 where 
the above mentioned gray values are in the range of [0, 256]). Also, these are the 
standard values for gray value image data. 

As to claim 1 1 , Luo teaches the criteria for classification as associated with the 
first region includes having an integer that exceeds a first threshold; and the criteria for 
classification as associated with the second region includes having an integer that is 
less than a second threshold (see column 9, lines 25-32, where the value of each pixel 
is proportional to its belief value and a global threshold is used to create a binary map, 1 
is considered a potential sky pixel and 0 is considered a non-sky pixel). 
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As to claim 12, Lou teaches the method of claim 11, wherein: the first threshold 
is 170 and the second threshold is 85. There is no real significant advantage for 
choosing the first threshold to be 170 and second threshold to be 85, therefore it 
obvious that the first and second threshold can be 170 and 85 respectively. 

9. Claim 17-19, 24, 26-30 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Luo et al (US Patent No.: 6,504,951) in view of Norton (US Patent 
No.: 5,912,994) further in view of Boskovitz et al (NPL Document titled "An Adaptive 
Neuro-Fuzzy System for Automatic Image Segmentation and Edge Detection"). 

As to claim 17, the combination of Lou and Norton do not teach the method of 
claim, wherein: the neighborhood of pixels is one of a three-by-three square of pixels, a 
five-by-five square of pixels, and a seven-by-seven square of pixels. 

Boskovitz teaches a method for multilevel image segmentation (page 247, [p][3], 
lines 1-2) that includes wherein: the neighborhood of pixels is one of a three-by-three 
(see Fig 6b) square of pixels and a five-by-five square of pixels (see Fig 6c). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to have added the method for multilevel image segmentation of 
Boskovitz to the method of defining a boundary of Luo as modified by Norton for 
segmentation of intensity image (page 261, [p][1], line 1) without "require any human 
expert intervention, nor any a priori information about the input image" (page 261, [p][1], 
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lines 20-23). Also, the use of 3x3 and 5x5 windows of pixels to process image data is 
extremely conventional and allows a large image to be processed in smaller, easier to 
process blocks. 

As to claim 1 8, note the discussion above, Boskovitz teaches the method 
wherein the pixel being considered is located at a center of the neighborhood of pixels 
(see Fig 6a). 

As to claim 19, note the discussion above, Boskovitz teaches the learning 
machine is a neural network (see Fig. 5); the neural network includes hidden nodes and 
gating nodes (see Fig. 5); and a gating node is associated with a corresponding hidden 
node, the gating node being configured to determine, based on a location of a pixel of 
being considered, a contribution the corresponding hidden node makes to an output of 
the neural network (page 263, [p][1]). 

As to claim 24, note the discussion of claims 1 and 19 above. 

As to claim 26, note the discussion of claims 17 and 18 above. 

As to claim 27, Boskovitz teaches the method of training the gating node to 
determine, based on the location of the pixel of being considered, a contribution the 
hidden node makes to an output of the neural network (page 253, [p]4]). 
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As to claim 28, Boskovitz teaches the method of training the hidden nodes to 
classify pixels as either associated with the first region or associated with the second 
region, wherein the training of the hidden nodes occurs during the training of the gating 
nodes (page 253, [p]4]). 

As to claim 29, Boskovitz teaches the method of wherein the neural network 
includes: input nodes (see input layer in Fig 5) configured to receive input information 
specifying the location of the pixel being considered and to provide the input information 
to the gating node. 

As to claim 30, Boskovitz teaches the method wherein the neural network 
includes: input nodes (see input layer in Fig 5) configured to receive input information 
specifying the color arrangement of the corresponding neighborhood of pixels and to 
provide the input information to the corresponding hidden node. 

10. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over Luo et al 
(US Patent No.: 6,504,951) in view of Norton (US Patent No.: 5,912,994) further in view 
Simon et al (Pub. No.: US 2004/0170337). 

As to claim 21 , Both Luo and Norton do not teach a wherein decontaminating 
produces an opacity mask, the method further comprising: constructing from the 
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identified pixels a probability mask; and combining the opacity mask and the probability 
mask. Simon teaches a method for enhancing an appearance of a face located in a 
digital image (page 2, [p][12], lines 3-4). Simon also teaches wherein decontaminating 
produces an opacity mask (generate alpha channel mask; page 6, [p][59], line 16-17), 
the method further comprising: constructing from the identified pixels a probability mask 
(page 6-7 [p][61], line 8); and combining the opacity mask and the probability mask 
(page 8, [p][72], lines 11-12). Therefore it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to have added the method for 
enhancing an appearance an image of Simon to the method of defining a boundary of 
Luo as modified by Norton to "enable the retouching of portrait without requiring skilled 
operator intervention to make and supervise the retouching correction" (page 2, [p][13], 
lines 3-5). 

As to claim 22, note the discussion above, Simmon teaches the method of 
combining the opacity mask and the probability mask includes multiplying the opacity 
mask with the probability mask (see page 7, [p][64], lines 4-5, where combining 
probability maps includes arithmetic multiplication). 

1 1 . Claim 23 is rejected under 35 U.S.C. 103(a) as being unpatentable over Luo et al 
(US Patent No.: 6,504,951 ) in view of Norton (US Patent No.: 5,912,994) further in view 
Mao (Pub No.: 2003/0063797). 
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As to claim 23, Lou teaches the method wherein the first region is a foreground 
(e.g. nonsky pixels such as the water and the boat in Fig. 12 A) of the image and the 
second region is a background (e.g. sky in figure 12 A) of the image. Note the 
discussion above, Norton teaches decontaminating includes excluding from the 
identified pixels a pixel that has no foreground colors (adjust border range, column 16, 
line 41-43). However, Both Lou and Norton do not teach changing colors of a pixel that 
includes both foreground and background colors so that the changed identified pixels 
include only foreground colors. Mao teaches a selection tool for defining a border area 
(page 1 , [p][0009] that includes teach changing colors of a pixel that includes both 
foreground and background colors so that the changed identified pixels include only 
foreground colors (change the color of a pixel in a target pixel map to foreground color) 
(page 4, [p][0048]). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to added the selection tool of Mao to the method of 
defining a boundary of Luo as modified by Norton to simplify "selection of complicated 
objects by providing a wide selection brush" (page 4, [p][0052], lines 2-4) and because 
each reference is for segmenting region of an image. 
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Conclusion 

The prior art made part of the record and not relied upon is considered pertinent 
to applicant's disclosure. 

Luo et al (US Patent No.: 7,062,085) is cited to teach a method for detecting 
subject matter regions in a digital color image. 

Okamina (US Patent No.: 6,778,701) cited to a feature extraction device for 
calculating features in pattern recognition that includes using a neural network as a 
classifier. 

Kim et al (Pub No.: US 2002/0102017) is cited to teach a method for sectioning 
an image in plurality of regions based on color and texture. 

Inquires 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Andrae S. Allison whose telephone number is (571) 
270-1052. The examiner can normally be reached on Monday-Friday, 8:00 am - 5:00 
pm, EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph Mancuso can be reached on (571) 272-7695. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Andrae Allison 
September 21, 2006 
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